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(54) Video information retrieval 

(57) A video Infomiatlon retrieval system comprises 
a client system having: means for Issuing a search re- 
quest in respect of desired video material; and means 
for accessing video material on the basis of a unlfomi 
resource locator (URL) and a SMPTE unique material 
identifier (UMID); a sen/er system having: access to one 
or more datat>ases cohtcUnlng metadata infomiatlon re- 
lating to a plurality of video material items, a UMID as- 
sociated with each video material Item and at least one 
URL associated with each UMID; means for receiving a 
search request from the client system and detecting one 
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or more video material Items for which metadata Infer- 
matton stored In at least one of the datBbase(s) substan- 
tiaiiy conresponds to the search request; means for sup- 
plying the metadata inf omnatlon, the URL and the UMID 
relating to the one or more detected video material items 
to the client system; and at least one video repository 
having: a video storage an^ngement storing video ma- 
terial and associated UMID data; in which the metadata, 
the URL and the UMID are communicated between the 
server and the client using a markup language having 
descriptors for data content. 
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Description 

10001] This Invention relates to video infomnation re- 
trieval. 

[0002] Video Images are a useful resource for enter- 
tainment and for dissemination of infbnnation. Digital 
video Images are also Increasingly being used In a wide 
range of multimedia applications. 
[0003] Thesheer volume of video information current- 
ly available to the user Is overwhelming with the exist* 
ence of many video libraries and archives each of which 
potentially stores millions of Images. These video ar- 
chives have a broad spectrum of users running different 
applications and requiring a range of services from pro- 
vision of subject-specific video clips for editing purposes 
to video on demand. In practical terms the video archive 
environment must allow users to run custom applica- 
tions which utilise a common database of video images 
and provide descriptive data related to the video Images 
to allow the user to make an Infomied choice of which 
media file to download. The generic term for the descrip- 
tive data associated with video images Is metadata. 
[0004] Computer database management systems 
have proved to be very effective for organising text and 
numeric data. The most widespread database manage- 
ment systems are known as "relational" databases. 
These systems collect data and organise it as a set of 
fomnally described tables from which data can be ac- 
cessed setectlvely and reassembled In a variety of ways 
without having to reorganise the data tables. The stand- 
ard user and appllcatton program Interface (API) to a 
relational database Is the structured query language 
(SQL) which can be used for simple interactive queries 
as well as for more extensive data gathering for use in 
compiling reports. 

[0005] A further example of an Infonnatlon manage- 
ment system Is a web search engine. The web search 
engine is Ideally suited for use in a multimedia environ- 
ment and has three baste components: 

■ A program known as a "spider" that goes to every 
page or representative pages on every web site that 
wants to be searchable and reads It. using hypertext 
links on each page to discover and read a site's oth- 
er pages. 

■ A program th at creates a master index from the pag- 
es that have been read. 

■ A program that receives a user's text-baeed search 
request, compares It to the entries in the master In- 
dex, and returns results to the user. 

[0006] Vktoo archives are of very limited value to the 
user unless there Is an information management system 
for images capable of delivering Images based on their 
specific content. This video information management 
system Is likely to require features used In database 
management systems as well as some of the function- 
ality of the web search engine. One difficulty Is that im- 



age and vkleo data require a much higher bandwidth 
than text-based Information. Downloading a video clip 
across a computer network can be very time consuming 
because of the large quantity of data Involved. In some 

5 cases the user may have to download and view several* 
vkleo dips In real time In order to find a clip with the 
required infonnatlon content. Thus It Is very important 
to provide the user with adequate Information about Im- 
ages In the archives prior to any download to increase 
the likelihood of the downloaded images meeting the us- 
er-specific requirements. Some users may be looking 
for video dips that can be used to Illustrate a particular 
feature or issue, for example, video segments showing 
a particular polttldan or dignitary. Other users might be 

IS searching for complete programmes and news Items re- 
lated to a specif to topic such as global wanning. It would 
also be advantageous to the user to have unrestricted' 
access to as many video archives as possible via a sin- 
gle video-speclfto search query. 

^ [0007] A typical prior-art video Infonnation retrieval 
system for use on the worid-wlde web is illustrated in 
Figure 1 . Video source material 1 0 Is Input as raw video 
Informatten 15 to an encoding and content-anaiysis 
module 20. The source material could be a digital or an- 

25 alogue video-cassette, an electronically stored digital 
video file or a broadcast signal fed directly via satellite 
The encoding and content-analysis module 20 takes the 
vkleo source material and produces digital copies It In. 
various alternative formats ranging from low btt-rf te ver- 

30 stens suitable for use on Internet browser plug-Ins such 
as ReaMdeoTM to high bit-rate broadcast quality 
MPEG2 Images. . 

[0008] On Input to the video archive system the ana- 
logue or digital source material Is subject to an automat- 

33 ed content-analysis process. This typically involves the 
use of local Intensity histograms, edge histograms, ge- 
ometrical shape analysis, face detection and on-screen 
text extraction to establish and log the content of each 
Image. The associated audio samples may be proc- 

^ eased for content using speech detectten algorithms.* 
Proprietaiy content-analysis software such as Virage's 
Videotogger'"' has b&en be used for this purpose. The 
result Is a video Index 25 which summarises the content 
of the video material. 

45 [0009] A video application server 30 stores the video 
Index 25 In an appropriate format so that It Is accessible 
to a web server 40. In addition the video application 
server 30 provides a flexible template system, handles 
client-queries and provides culmlnlstratlon tools. Clients 

30 60 running Internet browsers have access to the video 
Index via the web server 40. The clients may enter 
search tenns In a standard web search engine which Is 
Interfaced the video Index so that video material can be 
selectively retrieved on the basis of its logged content 

33 [0010] The encoding and content-analysis module 20 
outputs the digital video infomnatlon 65 across a distri- 
bution network. The digital video Information 65 Is avail- 
able for download to the dlents via a video server 50. 
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The video Index 25 Is used to search for and retrieve 
particular video dips required by users. 
[001 1 ] The invention provides a video information re- 
trieval system comprising: 

a dlent system having: 

means for issuing a search request In respect 
of desired video material; and 
means for apcessing video material on the ba- 
sis of a unifonn resource locator (URL) and a 
SMPTE unique material identifier (UMID); 

a server system having: 

access to one or more databases containing 
metadata Infonnation relating to a plurality of 
video material items, a UMID associated with 
each video material item and at least one URL 
assodated with each UMID; 
means for receiving a search request from the 
client system and detecting one or more video 
material items for which metadata Information 
stored in at least one of the database(8) sub- 
stantially corresponds to the search request; 
means for supplying the metadata Information, 
the URL and the UMID relating to the one or 
more detected video material items to the client 
system; 

and at least one video repository having: 

a video storage arrengement storing video n«- 
terial and associated UMID data; 

In which the metadata, the URL and the UMID are 
communicated between the server and the client using 
a maricup language having descriptors for data content. 
[0012] The Invention provides an Improved video in- 
formation retrieval system which (a) uses UMIDs to ac- 
cess video material, thereby providing a unique and 
platform- (or vendor-) independent Index to the video 
material, and (b) uses a maricup language having de- 
scriptors for data content as the transmission means for 
the search results, which means again that tlie commu- 
nication required for the video infonmation retrieval sys- 
tem can potentially be platform- cmd vendor-independ- 
ent as such maricup language files are potentially trans- 
missible via the generally available http port 80. 
[0013] Further respective aspects and features of the 
invention are defined In the appended claims. 
[0014] Embodiments of the Invention will now t>e de- 
scribed with reference to the accompanying drawings, 
throughout which nice parts are referred to by iilce refer- 
ences, and in which: 

Figure 1 schematically Illustrates a prior art video 
information retrieval system; 
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Figure 2 Is a schematic diagram of a video informa- 
tion retrieval system according to an embodiment 
of the present invention; and 
Figures 3 and 4 are schematic examples of the use 
5 of XML data structures. 

[MIS] Referring now to the drawings, Rgure 2^8 a 
schematic Illustration of a video infomtation retrieval 
system according to an embodiment of the Invention. A 

10 client 100 running a web bromer initiates a seardi re- 
quest 1 05 spedficalty directed to video material. The 
search is performed via a web search engine. The 
seardi engine communicates via a common gateway 
interim (CGI) on a server 110. The search engine con- 

IS verts the dient request to a datat>ase query 115 and the 
client request Is output as a signal 125 to a metadata 
database 1 30A or, if so required, to a series of databas- 
es (130A, 130B...) distributed across the internet. 
[0016] The main obstade in attempting to gain access 

20 to remote datc^ases of video material via the I ntemet is 
that in many cases dient and server machines will be 
separated by a ffrewa// or proxy server. Aflrewall Is aset 
of related programs, located at a networic gateway serv- 
er, that protects the resources of a private networic from 

25 users of other networtcs. By woridngdoseiy with a router 
program, a firewall filtere ail networic packets and. de- 
cides whetiier or not to forward them to their destin^^on. 
A proxy server which malces networic requests on t^ehalf 
of users may be Included In a firewall or woric closely 

30 witii It. Firewalls are generally able to distinguish one 
protocol from another. In the Transmission Control Pro- 
tocol/ Intemet Protocol (TCP/IP) architecture a specific 
port nuntbar is assigned to each common protocol and 
each request made using that protocol carries that 

S3 number. For example HTTP te assigned to poftao while 
RieTransfer Protocol (FTP) is assigned to port 2f. Most 
flrewails allow blocking of a spedf protocol by rejecting 
alt traffic sent on the port numt>er assodated with that 
protocol. Most firewalls are configured to let through traf- 

40 fic on port 80 which Is how HTTP requests from brows- 
ere get tiirough. Since each unblocked protocol poses 
a potential security threat, firewalls are generally set up 
to block most ports with the exception of port 60. As shall 
be explained batow, the interdiange between the client 

^ and the metadata database according to entbodfrnents 
of the Invention, is In a maricup language that has de- 
scriptors for data content such as XML. Since XML Is 
text-based, aclvantage can be taken of HTTP port 60 to 
deploy an Intemet-wide video archive search facility. 
so HTTP alone would not be sufficient to Implement 
seardies on remote datat^ases of video material across 
multiple platforms because It lacks a single standard for- 
mat for representing queries. Because XML is a plat> 
fonm-neutral data representation, it can be used on top 
93 of IHTTP to serialise (teta into a transmissible fomn that 
is easily decoded on any platform. Tills Is the basis on 
which remote procedure call (RPC) protocols such as 
Microsoft^ Simple Object Access Protocol (SOAP^) 
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operatas. RPCs are specialty designed to provide ac- 
cess to computer program objects resident on machines 
that are distrbuted across ttie Intemet 
[0017] in a video retrieval system designed for de- 
ployment across the Intemet there will be no central 
management of the video archives, and therefore It is 
very ^portant to be able to uniquely and unambiguous- 
ly Identify each video cl^ that Is accessible to the user. 
The metadata database 130 uses the SMPTE UMID to 
relate the stored metadata to the partlcuiar video mate- 
rial from which It was generated. 
[OOiq The UMiO Is described in the March 2000 Is- 
sue of the "SMPTE Journal'. An "extended UMID' com- 
prises a first set of 32 bytes of "basic UMID' and a sec- 
ond set of 32 t>yte8 of "signature metadata". 
The basic UMID has a key-length-value (KLV) stnicture 
and It comprises: 

■ A 12-byte Universal Label or icey which Identifies 
the SMPTE UMID itself, the type of material to 
which the UMID refers, it also defines the methods 
by which the globally unique Material and locally 
unique Instance numbers (defined below) are cre- 
ated. 

■ A 1 -byte length value which specifies the length of 
the rernaining part of the UMID. 

■ A 3-byte Instance numt>er used to distinguish be- 
tween different "Instances" or copies of material 
with the same Material number. 

■ A 16-byte Material numt>er used to identify each 
clip. A Material number Is provided at least for each 
shot and potentially for each Image frame. 

The signature metadata oomprtees: 

■ An 8-byte time-date code identifying the time of cre- 
ation of the 'Content Unlf to which the UMID ap- 
plies. The first 4-bytes are a Universal Time Code 
(UTC) based component. 

■ A 12-byte value which defines ttie (GPS derived) 
spatial co-ordinatea at the time of Content Unit cre- 
ation. 

■ 3 groups of 44}yte codes which comprise a country 
code, an organisation code and a user code. 

[001 9] The metadata databases 1 30 contain data de- 
scribing the content of video material. The metadata in- 
cludes location infomiation for the video images to 
which it corresponds, such as a uniform resource locator 
(URL). The URL for a video dip Is associated with the 
UMID Identifier and an additional timecodecan be used 
to obtain particular still images from a given clip. The 
metadata also Includes analysis data from post- 
processing of the Image signal such as sub-shot seg- 
mentation infonnation and Informatiori c^out an Image 
frame called a representative keystamp (RKS) which 
gh/es a visual Indication of the predominant overall con- 
tents of each shot or sub-shot. 
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{0020] Proprietary oontent-extraction tools such as' 
Virago's Videologger^ can be used to obtain descrip- 
tive infonnation about the component ^'objects' in each 
video such as people, buildings, cities, the topic or 
s event to which the clip relates, actors names and details 
of the ownership rights of thefootage.ThecontentHndex 
for each video clip Is stored as metadata. The metadata 
can be stored In the databases 130 in any forma^ 
[0021] As Illustrated in Rguro 2, the server 1^0 re- 
io spends to the client search request 1 05 by returning an 
XML file 155 containing metadata for the yideo clips 
which match tiie user's search request. XML is an ex-, 
ample of a martcup language. Aitiiough XML is the pre- 
ferred maricup language for Interchange of data be- 
ts tween the client and the databases 1 30, any maricup lan- 
guage that has descriptors for data content could be 
used. Maricup languages are computer programming 
languages In which document stnjctures are Indicated 
in the same stream as the text. Mariners like < and > 
20 divide documents Into elements arid attributes. Ele- 
ments are containers hoM that hoM content and possibly 
ottier elements inside them In a hierarchy. Attributes 
provide additional information about a particular ele- 
ment. Elements and attributes are specified by tags en- 

29 dosed In < and >. A start tag indudes the element name 
and the names and values of the attributes while an end 
tag Is marked by a forward-slash character and includes 
only the name of the element corresponding to the start 
tag that It matches. The syntax is as follows: 

30 start tag: <iel0fiMfifMBni# ailributoNme s 'attribiito- 

twnbteiudedhemlnbodyefmlmnent 
End tag : </efmn9ntNmm6> 
[0022] Hypertext markup language (HTML) Is the lan- 

35 guage of the worid-wlde web and its tags comprise a 
pre-defined and no n -extensible set that describe docu-* 
ment fonnat i.e. how the contents of a document should 
be displayed. XML has tags which define an information 
structure by describing document content rather than 

40 document fonnat it allows developers to extend the set 
of tags used and to create ttieir own vocabulary for de- 
scribfng infonnation. A "schema" Is a set of rules tiiat 
describes a give n dasa of XML documents. The schema 
defines the elements that can appear and their corre- 

45 spending attributes. It also defines the hierarchical 
structure by specifying which elements are child ele- 
ments of others, the order In which chlM elements ap- 
pear and the number of child elements. XML is a sim- 
plified subset of Its parent maricup language, Standard 

50 Generalised Maricup l^guage (SGML). XML Is de- 
signed to aBow the exchange Infomnation between a 
host of different applications running on different types 
of computers without repeated conversion to proprietary 
file fomiats. Although XML Is the preferred language, 

5S any extensible markup language with the fadlity for data 
description tags could be used as a file format for data 
storage In the metastore. 

[0023] An example portion of an XML file that might 
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be used in embodiments of the invention is shown in 
Flpure 4. The <medla> tag occurs at the top level of the 
hierarchy and contains at the next ieyei down, the 
"metadata objecte" element and the "metadata trades" 
element The child elements of the metadata objects are 
shown as elements for person, place and topic, each of 
which has. an "hreT attribute. This attribute provides a 
link to an image associated writh the respective metada- 
ta object. The body of each element contains the infor- 
mation itself, for example there are person elements in 
Rgure 4 that mark the names of Blii Clinton and Nelson 
Mandela. The metadata object elements mark text- 
based descriptions of objects that appear in the images 
while the metadata tracks provide an Index to the sut>set 
of images of a clip in which the particular metadata ot^ 
Ject associated with the metadata track feeUures. The 
UMID is included as a child element of the metadata 
tracks. The advantage of explteitly providing an index to 
the subset of Images In which an object appears Is that 
rather than downloading an entire video clip with which 
the object is associated, only the subset of Images and 
the associated audio In which the metadata object ap- 
pears need be downloaded from the video store. This 
reduces download time and saves bandwidth. The fuii 
dip can also be downloaded if so requb^ed. 
[0024] Figure 5 shows the hierarchical structure of the 
XML metadata f He of Figure 3. The media tag 200 is at 
the top level of the hierarchy. The metadata objects 220 
and the metadata tracks 21 0 are both child elements of 
the media element 200. Each metadata object has acor- 
responding metadata track partner. This is Illustrated by 
the person element 230A whbh corresponds to the per- 
son track 230B. The UMID elements 240 are at the low- 
emiost level of the hierarchy in this case. 
[0025] The fact that the interchange between the cli- 
ent and the database is in XML provides advantages 
over the prior-art systems. In parttoular. the XML inter- 
face between dient and database allows complex que- 
ries to be constmcted using XML query language. The 
software Interfaces between the client and the metas- 
tore are Independent of the particuiar data schema used 
by the customer whtoh means that the customer has the 
freedom to design and use his own specific business 
schema In conjunction with the video material datcdsase 
according of the invention. The video Infonmation re- 
trieval system of the present invention also allows for 
easy Integration of proprietary video content-extraction 
tools and database systems from other vendors. 
[0026] The XML file 155 will Indude URLs for low 
bandwidth and full bandwidth vereions of the video dips. 
The user may require full bandwidth video material for 
use with high-end equipment orto indude in a television 
broadcast. Low bandwkfth video material may be re- 
quired by the userforviewlngon low-end equipment for 
editing puiposes or for transmission across computer 
networics. The XML file will also provide links to still im- 
ages such as the representative keystamp (RKS) imag- 
es for each of the video dips highlighted by the search 



query. The RKS images are located by a CGI script host- 
ed by a w^ server which takes the UMID and the time- 
code as parametere. 

[0027] The XML file ts converted to HTML and dls- 
5 played in the cHents browser. The user at the dient com- 
puter makes a decision as to which video material to 
download on the basis of the metadata provided. To 
download the vMeo material the user initiates a client 
request 165 whbh Is dlreded to the appropriate video 
10 server using the URL and UMID Infonnation contained 
in the XML nie 155. 

[0028] Although the metadata can be stored In the da- 
tabases 1 30 In any fomiat, because the exchange of da- 
ta between the databases 130 and the dient 100 is In 

IS XML. it may also be convenient to store metadata in hi- 
erarchical fomfiats In the databases 1 30 using XML The 
databases 130 could use an object database to store 
the XML metadata files. The hierarchical structure of 
XML means that it is more efficient to store complex 

so XML files In an object database rather than a relational 
database. The XML Is parsed into object structures prior 
to being stored in the object database. The use of the 
object database has the advantage that the information 
is stored in a format whtoh makes it easy to access ei- 

S5 ements and attributes rapkliy without the requirement of 
toading and parsing of a sequential file. 



Claims 

30 . 

1 . A video Information retrieval system comprising: 
a client system having: 

35 means for issuing a search request in re- 

spect of desired video material; and 
means for accessing video material on the 
basis of a uniform resource locator (URL) 
and a SMPTE unique material Identifier 

^ (UMID); 

a server system having: 

access to one or more databases contaln- 
^ ing metadata Information relating to a plu- 

rality of video material items, a UMID as- 
sodated with each vkleo material Item and 
at least one URL assodated with each 
UMID; 

^0 means for receiving a search request from 

the dient system and detecting one or 
more video material items for which meta- 
data Infonnation stored In at least one of 
. the database(s) substantially conresponds 

55 to the search request; 

means for supplying the metadata informa- 
tion, the URL and the UMID relating to the 
one or more detected video material Items 
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to the client eystem; 

and at least one video repository having: 

a video storage arrangement storing video 
nnaterial and associated UMID data; 

In which the metadata, the URL and the UMID 
are communicated between the server and the cli- 
ent using a markup language having descriptors for 
data content. 

2. A system according to dalm 1 , (n which the search 
requests are communicated between the server 
and the client using a markup language having de- 
scriptors for data content. 

3. A system according to claim 1 or claim 2, in which 
the database stores metadata in a hierarchical rep- 
resentation using a markup language having de- 
scriptors for data content. 

4. A system according to any one of ciainrts 1 to 3, in 
which the markup language Is an extensible markup 
language (XML). 

5. A system according to any one of the preceding 
claims, in which the ciient and the server communi- 
cate via http port 80. 

6. A system according to any one of the preceding 
claims, in which the server system Is operable to 
supply URLs to the client system tor accessing the 
video material in a broadcast-quaiity representa- 
tion. 

7. A system according to any one of the preceding 
claims, in which the server system is operable to 
supply URLs to the client system for accessing the 
video material in a sub-broadcast-qualtty represen- 
tation. 

8. A system according to any one of the preceding 
claims, In which the server system Is operable to 
supply URLs and video timecodes to the client sys- 
tem for accessing single Images representative of 
the content of the video material. 

9. A system according to any one of the preceding 
claims, In which the server, the client and the video 
repository communicate via the world wide web. 

10. A video infomiatlon server having: 

access to one or more databases containing 
metadata infomr^atlon relating to a plurality of 
video material itents. a SMPTE unique material 
identifier (UMID) associated with each video 
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material item and a uniform resource locator 
(URL) assodatad with each UMID; 
means for receMng a search request from a cli- 
ent system and detecting one or more video 

5 material items for which metadata Infonnation 

stored in at teast on of the databa8e(s) substan- 
tially corresponds to the search request; 
■ means for supplying the metadata Infonmation, 
the URL and the UMID relating to the one or 

10 more detected video material items to the client 

system using a rnariaip language having de-* 
scriptors for data content 

11. Avideo Information retrieval client system compris- 
f« Ing: 

means for Issuing a search request to a video 
infonnation server system In respect of desired 
video material; 

^ means for receiving search results from the 

server system comprising at least a uniform re- 
source locator (URL) and a SMPTE unique ma- 
terial Uentifler (UMID); and 
means for accessing video data from a video 

S5 repository on the basis of the URL and the 

UMID data; 

In which the metadata, the U RL and the UM ID 
are communicated between the server and the cii- 
30 ent using a mari(up language having descriptors for 
data content 

12. Amethod of video information retrieval using a serv- 
er system having accese to one or more databases. 

95 containing metadata infonnation relating to a plural- 
ity of video material items, a SMPTE unique mate- 
rial identifier (UMID) associated with each video 
material item and a URL associated with each 
UMID; 

40 the method comprising the steps of: 

a client eystem Issuing a search request in re- 
spect of desired video material; 
the server system recehring the search request 
^ from the ciient system and detecting one or 

more video material items for which metadata* 
infonnation stored In at least one of the data- 
base(s) substantially corresponds to the search 
request; and 

so the sender system supplying the metadata In- 

formation, the URL and the UMID relating to the 
one or more detected video material Items to 
the.cllent system using a markup language hav- 
ing descriptors for data content; 

» the dlent system accessing video material on 

the basis of the unifonn resource locator (URL) 
from a video repository having a video storage 
an-angement storing video material and asso- 
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dated UMID date. 

13. A method of video Information retrieval, the method 
being subetantlalty as hereinbefore described with 

^ reference to Rgures 2 to 4 of the accompanying 
drawings. 

14. Computer software having program code for cany- 
Ing out a method according to dalm 12 or dalm 13. 

18. A data providing medium by which computer soft- 
ware according to claim 14 le provided. 

16. A medium according to dalm 15, the medium being 
a transmission medium. 

1 7. A medium according to dalm 1 5, the medium t>elng 
a storage medium. 
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